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Objectives 


The  focus  of  the  project  is  to  develop  control  theoretic  models  with  supporting  decision  making  methods  to 
address  the  problem  of  strategic  decision  making  for  military  operations  in  an  environment  that  is  strongly 
influenced  by  the  presence  of  interacting  subcultures.  The  specific  directions  of  the  project  are:  1 )  Modeling 
of  social  behaviors  that  address  incomplete  information  and  indeterminate  models  for  uncertainty  manage¬ 
ment,  2)  Robust  decision-making  based  on  incentive  design,  optimization,  and  learning,  and  3)  Constructive 
computational  tools. 

Research  Directions 

The  research  effort  was  focused  on  four  areas,  1 )  forecasting  enemy  intent:  the  development  of  construc¬ 
tive  forecasting  algorithms  that  provide  statistically  consistent  forecasts  in  the  absence  of  underlying  pro¬ 
cess  models;  2)  dynamic  adaptation  in  coordination  games:  the  analysis  of  learning  rules  based  on  over¬ 
reactionary  behaviors;  3)  analysis  of  rational  social  learning  in  networks:  analysis  of  belief  propagation  in 
social  networks  in  various  network  topologies;  and  4)  distributed  multi-agent  optimization:  analysis  and  de¬ 
velopment  of  distributed  optimization  methods  in  multi-agent  networks  that  operate  with  local  information 
and  computations. 

Accomplishments 

Forecasting  enemy  intent:  The  problem  is  to  develop  a  statistically  consistent  forecast  of  a  recurring 
event.  Prior  work  derived  “universal”  forecasting  algorithms  that  guaranteed  statistical  consistency  regard¬ 
less  of  the  underlying  process  generating  the  recurring  event.  A  severe  drawback  of  these  algorithms  is  that 
they  are  essentially  impossible  to  implement.  Our  work  has  developed  a  forecasting  algorithm  that  trades 
off  computationally  feasibility  with  universality.  The  forecasting  method  provides  statistically  consistent 
forecasts  for  a  large  class  of  opponent  models  while  having  very  modest  computational  requirements. 

Our  work  also  broadened  the  scope  to  forecasting  performance  of  competing  policies  in  unknown  reac¬ 
tionary  environments.  We  have  developed  an  algorithm  that  adaptively  switches  between  alternate  policies 
in  an  effort  to  determine  which  is  most  effective.  The  forecasting  method  assures  that  the  resulting  perfor¬ 
mance  asymptotically  approaches  that  of  the  best  policy. 

This  work  has  been  accepted  for  journal  publication,  conference  publication,  and  presentation. 

Dynamic  adaptation  in  coordination  games:  Our  work  has  shown  that  “dynamic  adaptation”  can  enable 
qualitative  changes  of  behavior  when  compared  to  more  conventional  learning  models.  In  this  work,  we 
investigate  dynamic  adaptation  for  coordination.  A  simple  example  is  as  follows.  There  are  two  competing 
conventions,  A  and  B.  A  group  of  agents  is  in  equilibrium  when  all  follow  either  convention  A  or  con¬ 
vention  B.  However,  convention  A  is  more  efficient  in  that  all  agents  would  achieve  a  greater  reward.  In 
conventional  distributed  reinforcement  learning,  convention  B  becomes  a  stable  attractor.  In  our  “dynamic 
adaptation”  version  of  reinforcement  learning,  actions  are  reinforced  by  both  rewards  and  rate  of  change  of 
rewards.  The  result  is  that  efficient  convention  A  becomes  the  only  stable  attractor  —  even  if  only  one  agent 
uses  dynamic  adaptation. 

Follow  on  work  has  focused  on  specializing  the  general  methodology  as  a  predictive  device  for  social 
network  formation  and  for  communication  network  formation  with  constraints  on  the  lengths  of  communi¬ 
cation  chains. 

This  work  has  been  presented  as  a  conference  publication  and  is  under  preparation  for  journal  submis¬ 
sion. 
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Analysis  of  rational  social  learning  in  networks:  In  social  networks,  an  important  issue  is  to  understand 
the  propagation  of  information  through  the  network,  i.e.,  individuals  learning  from  others’  actions  and  in¬ 
formation  made  available  through  other  channels  (such  as  media).  In  the  context  of  communities  of  social 
networks,  much  of  the  learning  comes  in  the  form  of  word-of-mouth  or  communication  between  neighbors 
and  acquaintances.  This  phenomenon  is  sometimes  referred  to  as  social  learning. 

Although  there  is  a  large  literature  in  economics  on  social  learning,  this  literature  does  not  focus  on  the 
implications  of  the  social  network  topology  and  interaction  structure  on  the  information  dissemination  and 
belief  formation.  Most  of  the  work  relies  on  the  unrealistic  assumption  of  perfect  observability  of  the  entire 
history  of  actions.  Our  interest  in  this  work  is  to  determine  how  outcomes  of  social  learning  is  sensitive  to 
observation  structures,  social  network  topology,  or  deliberate  manipulation. 

In  recent  work,  we  presented  an  analysis  of  social  learning  on  a  line  network  when  individuals  have 
access  to  a  “noisy’'  measurement  of  a  fixed  unknown  variable  and  only  observe  the  action  of  their  immediate 
neighbor.  The  resulting  structure  is  the  so-called  Bayes-Nash  equilibrium.  Despite  the  simplicity  of  this 
environment,  the  evolution  of  beliefs  is  substantially  different  from  the  typical  models  of  social  learning  in 
the  game  theory  literature.  We  characterized  the  behavior  of  asymptotic  learning  in  terms  of  threshold  values 
that  evolve  deterministically.  Individual  actions  are  fully  determined  by  the  value  of  their  signal  relative  to 
these  thresholds.  We  characterized  conditions  on  the  private  information  of  the  individuals  under  which  they 
can  learn  the  true  value  of  the  variable.  In  more  recent  work,  we  extended  our  analysis  to  general  network 
topologies  and  developed  conditions  on  the  network  topology  that  lead  to  ’’asymptotic  learning”.  We  also 
presented  rate  of  convergence  results  that  highlight  the  dependence  on  the  observation  window  length. 

This  work  has  been  accepted  for  conference  publication  and  presentation,  and  two  journal  publications 
are  being  prepared  for  submission. 

Distributed  Multi-Agent  Optimization:  We  consider  a  cooperative  multi-agent  environment,  where  each 
agent  has  a  local  cost  function  and  constraint  structure,  which  is  known  to  that  agent  only.  We  introduce 
a  new  framework  for  the  development  and  analysis  of  distributed  asynchronous  optimization  algorithms  to 
optimize  the  global  objective  of  the  sum  of  the  individual  local  cost  functions.  For  this  purpose,  we  use  first- 
order  or  subgradient  methods  which  are  gaining  popularity  in  the  optimization  community  as  low-overhead 
alternatives  to  interior  point  methods  for  large-scale  convex  optimization  that  lend  themselves  to  distributed 
implementations.  We  consider  a  distributed  computation  model,  whereby  every  agent  generates  and  main 
tains  estimates  of  the  optimal  solution,  and  communicates  them  directly  or  indirectly  to  the  other  agents, 
potentially  with  delays.  Each  agent  updates  their  estimate  by  combining  it  with  the  estimates  received  from 
the  others  agents  (if  any)  and  by  using  the  gradient  (or  subgradient)  information  of  the  local  cost  function. 
In  recent  work,  we  have  established  convergence  results  and  convergence  rate  estimates  for  this  model.  Our 
results  are  novel  in  the  sense  that  they  capture  the  tradeoffs  associated  with  the  suboptimality  of  the  solutions 
and  the  computational  complexity  of  the  algorithms. 

This  work  has  been  presented  in  conferences  and  has  appeared  as  journal  publications. 

We  have  also  pursued  game  theoretic  approaches  to  distributed  multi-agent  optimization.  Most  work 
in  learning  in  games  assumes  that  the  same  conflict/game  is  infinitely  repeated.  In  stochastic  games,  the 
conflict  depends  on  an  underlying  state,  and  the  state  evolution  depends  on  the  actions  taken  by  the  players 
in  the  game.  We  have  investigated  the  special  case  where  the  “state”  is  the  immediately  preceding  action 
of  an  agent,  i.e.,  the  set  of  possible  actions  at  one  stage  is  constrained  by  the  action  taken  in  the  previous 
stage.  We  have  derived  learning  algorithms  with  guaranteed  convergence  properties  for  special  classes  of 
multiagent  games  with  this  state  dependency.  We  also  have  investigated  the  issue  of  agent  information. 
Much  work  on  learning  assumes  agents  can  measure  the  actions  taken  by  other  agents.  We  have  investigated 
the  more  natural  assumption  where  agents  only  measure  their  own  realized  reward  and  need  not  have  a 
closed  form  expression  of  their  own  utility  functions.  We  have  established  convergence  results  for  certain 
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classes  of  games  under  such  constraints. 

This  work  has  been  accepted  for  journal  publication,  conference  publication,  and  presentation. 

Personnel  Supported  by  and/or  Associated  with  Project 

Faculty: 

•  Jeff  S.  Shamma,  UCLA  &  Georgia  Institute  of  Technology 

•  Munther  A.  Dahleh,  MIT 

•  Asuman  Ozdaglar,  MIT 
Students: 

•  George  Chasparis,  UCLA 

•  Jason  Marden,  UCLA 

•  Ibrahim  Al-Shyoukh,  UCLA 

•  Ilan  Lobel,  MIT 


Publications  Submitted  and/or  Accepted 

•  S.  Mannor,  J.S.  Shamma,  and  G.  Arslan,  “Online  calibrated  forecasts:  Memory  efficiency  vs  uni¬ 
versality  for  learning  in  games”,  Machine  Learning ,  Special  issue  on  “Learning  and  Computational 
Game  Theory”,  May  2007,  pp.  77  -  115. 

•  J  R.  Marden,  G.  Arslan,  and  J.S.  Shamma,  “Connections  between  cooperative  control  and  potential 
games  illustrated  on  the  consensus  problem”,  2007  European  Control  Conference ,  Kos,  Greece. 

•  G.  Chasparis  and  J.S.  Shamma,  “Distributed  dynamic  reinforcement  of  efficient  outcomes  in  multia¬ 
gent  coordination”,  2007  European  Control  Conference ,  Kos,  Greece. 

•  D.  Acemoglu,  M.  Dahleh,  I.  Lobel  and  A.  Ozdaglar,  “Bayesian  Learning  in  Social  Networks”,  LIDS 
Working  Paper  #2780,  2008,  submitted  for  journal  publication. 

•  D.  Acemoglu  and  A.  Ozdaglar,  “Competition  in  Parallel-Serial  Networks,”  selected  for  fast-track  to 
IEEE  Journal  on  Selected  Areas  in  Communications ,  special  issue  on  Non-cooperative  Behavior  in 
Networking,  vol.  25,  no.  6,  pp.  1 180-1 192,  August  2007. 

•  D.  Acemoglu,  R.  Johari,  and  A.  Ozdaglar,  “Partially  Optimal  Routing,  IEEE  Journal  on  Selected 
Areas  in  Communications ,  special  issue  on  Non-cooperative  Behavior  in  Networking,  vol.  25,  no.  6, 
pp.  1148-1160,  August  2007. 

•  A.  Ozdaglar  and  R.  Srikant,  “Pricing  and  Incentives  in  Communication  Networks”,  chapter  in  Al¬ 
gorithmic  Game  Theory ,  Noam  Nisan,  Tim  Roughgarden,  Eva  Tardos,  and  Vijay  Vazirani,  editors, 
Cambridge  University  Press,  2007. 

•  J.R.  Marden,  H.P  Young,  G.  Arslan,  and  J.S.  Shamma,  “Payoff  based  dynamics  for  multiplayer 
weakly  acyclic  games”. 
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-  SIAM  Journal  on  Control  and  Optimization ,  special  issue  on  “Control  and  Optimization  in  Co¬ 
operative  Networks*’,  to  appear  2009. 

-  46th  IEEE  Conference  on  Decision  and  Control ,  New  Orleans,  Louisiana,  2007. 

•  I. A.  Al-Shyoukh  and  J.S.  Shamma,  “Switching  Supervisory  Control  Using  Calibrated  Forecasts”, 

-  46th  Conference  on  Decision  and  Control ,  New  Orleans,  LA,  December,  2007. 

-  IEEE  Transactions  on  Automatic  Control ,  to  appear  2009. 

•  LA.  Al-Shyoukh,  Online-information-based  learning  and  decision  making  under  uncertainty ,  Ph.D. 
Dissertation,  UCLA,  June  2007. 

•  J.R.  Marden,  Learning  in  Dirge-Scale  Games  and  Cooperative  Control ,  PhD  Dissertation,  UCLA, 
June,  2007. 

•  I.  Lobel,  D.  Acemoglu,  M.  Dahleh,  and  A.  Ozdaglar,  “Social  Learning  with  Partial  Observations,” 

-  46th  Conference  on  Decision  and  Control ,  New  Orleans,  LA,  December,  2007. 

-  Workshop  on  interdisciplinary  systems  approach  in  performance  evaluation  and  design  of  com¬ 
puter  and  communication  systems ,  France,  2007. 

•  A.  Nedic  and  A.  Ozdaglar,  “Distributed  Subgradient  Methods  for  Multi-agent  Optimization,” 

-  46th  Conference  on  Decision  and  Control ,  New  Orleans,  LA,  December,  2007. 

-  IEEE  Transactions  on  Automatic  Control ,  January  2009,  pp.  48-61 . 

•  A.  Ozdaglar,  ’’Constrained  Consensus  and  Alternating  Projections,”  45th  Annual  Allerton  Conference 
on  Communication ,  Control ,  and  Computing ,  2007,  invited  paper  and  presentation. 

•  G.  Chasparis  and  J.S.  Shamma,  “Efficient  network  formation  by  distributed  reinforcement”,  accepted 
for  publication,  47th  IEEE  Conference  on  Decision  and  Control ,  2008. 

•  G.  Chasparis,  “Distributed  Learning  and  Efficient  Outcomes  in  Uncertain  and  Dynamic  Environ¬ 
ments,”  PhD  Dissertation,  University  of  California,  Los  Angeles,  Department  of  Mechanical  &  Aerospace 
Engineering,  Los  Angeles,  CA,  May  2008. 


Interactions/Transitions 

•  Participation/presentations  at  meetings,  conferences,  seminars,  etc. 

-  “Learning  in  games  and  feedback  control”,  presented  at  the  Caltech  Workshop  on  Integrated 
Control ,  Estimation  and  Communication ,  Pasadena,  CA,  March  3 1  —April  2,  2005. 

-  “Calibrated  forecasts:  Efficiency  versus  universality”,  presented  at  the  The  I6th  International 
Conference  on  Game  Theory  at  Stony  Brook  University ,  Stony  Brook,  NY,  July  11-15,  2005. 

-  J.S.  Shamma,  “Control  theoretic  methods  for  competition  and  evolution”,  Caltech  Workshop  on 
Learning  and  Information  in  Games  and  Control ,  Pasadena,  CA,  March  22-23,  2006. 

-  J.S.  Shamma,  “Learning  in  Games  and  Feedback  Control”,  Economic  Theory  Workshop ,  Uni¬ 
versity  of  Wisconsin,  September  15,  2006. 
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-  A.  Ozdaglar,  “Economic  Incentives  and  Traffic  Engineering  in  Networked-Systems,”  MIT  Infor¬ 
mation  Technology  Conference  on  “Next  Generation  Computing  and  Wireless  Networks”,  April 
2006. 

-  A.  Ozdaglar,  “Competition  and  Efficiency  in  Congested  Networks”,  Radcliffe  Exploratory  Sem¬ 
inar  on  Dynamic  Networks:  Behavior,  Optimization  and  Design,  Radcliffe  Institute,  Harvard 
University,  October  21-22,  2006. 

-  J.R.  Marden,  H.R  Young,  G.  Arslan,  and  J.S.  Shamma,  “Convergence  Results  for  Payoff-Based 
Dynamics  in  Weakly  Acyclic  Games”,  presented  at  18th  Annual  International  Conference  on 
Game  Theory ,  Stony  Brook,  NY,  July  9-13,  2007. 

-  G.  Chasparis,  G.  and  J.S.  Shamma,  “Distributed  Dynamic  Reinforcement  of  Efficient  Outcomes 
in  Multiagent  Coordination,  presented  at  European  Control  Conference ,  Greece,  Kos,  July  2-5, 
2007. 

-  J.R.  Marden,  G.  Arslan  and  J.S.  Shamma,  “Connections  Between  Cooperative  Control  and  Po¬ 
tential  Games  Illustrated  on  the  Consensus  Problem”,  presented  at  Proceedings  of  the  2007 
European  Control  Conference ,  Kos,  Greece,  July,  2007. 

-  G.  Chasparis  and  J.S.  Shamma,  “The  Emergence  of  Efficient  Social  Networks  by  Dynamic  Re¬ 
inforcement,”  presented  at  4th  Lake  Arrowhead  Conference  on  Human  Complex  Systems ,  April 
25-29,  2007. 

-  G.  Chasparis,  “Dynamic  Reinforcement  of  Efficient  Outcomes  in  Multi-Agent  Coordination”, 
Social  and  Information  Sciences  Laboratory,  California  Institute  of  Technology,  June  2007. 

-  I.  Lobel,  “Social  Learning  with  Partial  Observations,”  presented  at  18th  Annual  International 
Conference  on  Game  Theory ,  Stony  Brook,  NY,  July  9-13,  2007. 

-  A.  Ozdaglar,  “Price  and  Capacity  Competition,”  presented  at  the  The  Economics  of  the  Software 
and  Internet  Industries  Conference ,  Toulouse;  January  2007. 

-  A.  Ozdaglar,  “Distributed  Multi-agent  Optimization,”  presented  at  the  22nd  European  Confer¬ 
ence  on  Operational  Research:  Nonsmooth  Optimization ,  Theory \  Algorithms ;  and  Applications, 
Prague;  July  2007. 

-  Asuman  Ozdaglar  &  Jeff  S.  Shamma  presented  works  at  the  Caltech  Microeconomic  Dynamics 
Workshop,  May  2008. 

-  Asuman  Ozdaglar  &  Jeff  S.  Shamma  were  co-organizers  of  the  Workshop  on  Frontiers  of  Game 
Theory  and  Networked  Control  Systems,  October  2008. 

-  Jeff  S.  Shamma  presented  at  the  AFOSR  Workshop  on  Mathematical  Modeling  of  Socio-Cultural 
Processes,  Santa  Barbara,  CA,  September  2008. 

-  Jeff  S.  Shamma  presented  the  Jones  Seminar  on  Science,  Technology,  and  Society,  Dartmouth 
College,  May  2008. 

•  Consultative  and  advisory  functions  to  other  laboratories  and  agencies. 

-  Munther  A.  Dahleh:  Sandia  National  Labs 

-  Jeff  S.  Shamma:  Air  Force  Scientific  Advisory  Board. 

•  Transitions:  None  to  report. 
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New  discoveries,  inventions,  or  patent  disclosures 

None  to  report. 

Honors/Awards 

•  Munther  Dahleh: 

-  IEEE  Fellow. 

•  Asuman  Ozdaglar: 

-  NSF  CAREER  Award. 

-  MIT  Class  of  ’43  Career  Development  Chair,  May  2006. 

-  American  Automatic  Control  Council  Donald  R  Eckman  Award,  2008. 

•  Jeff  Shamma: 

-  IEEE  Fellow. 

-  Julian  T.  Hightower  Chair  in  Systems  and  Control,  Georgia  Institute  of  Technology. 

•  Jason  Marden: 

-  Outstanding  graduating  mechanical  engineering  PhD  student,  UCLA,  2007. 
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